If you want to know what's really going on, shine a light on it...
As I mentioned in the prior post, there's a powerful approach for reducing complexity that allows a significant number of objects to be removed from your system, while decreasing maintenance and subsequently improving time-to-market for new features and functionality.
Sounds almost too good to be true! Fortunately, not only is it true, but it is likely that you already leverage this technique in a limited fashion.
The Way It Used To Be
To fully describe this technique, I would first like to take you back in time. Back to the misty, barely remembered shadows of antiquity. Back to the dawn of software as we know it, maybe five years ago.
Back then, we were still connecting our systems to one-another using outdated techniques like proprietary file formats, serialized classes, fixed-length fields, proprietary web services, custom objects, specialized record formats, etc.
Back then, we didn't understand that this created tight dependencies between our systems. For example, if a proprietary file format needed to be changed, every system consuming that file format would also need to change - and change in a tightly synchronized fashion, or you would end up supporting multiple legacy formats.
Back then, we didn't understand that the dependencies between our systems were holding us back. That we were all afraid of changing our system interfaces and triggering the inevitable firestorm of flaming dominoes, with each one falling and igniting the next in a roaring inferno of expensive and disruptive change.
The Way It Is Now
We have learned a lot since then. Now, we know better. We connect our systems to one another using universal and standardized formats capable of accommodating any data structure. Today, we know how to use JSON and XML to connect our systems in a clean way that readily accommodates change.
Managing and communicating lengthy and complex inter-system communication and object standards are a thing of the past. It has all been reduced to a simple schema that is easily shared, and even can be largely ignored - except for just the pieces that you are interested in, or that are absolutely necessary.
In a matter of minutes, you can now spin up a REST interface and have the entire world happily making calls into your system with minimal documentation or fuss.
So What?
Why is this important? First, its not completely true:
Many companies continue to use legacy methods of connecting their systems, both externally and internally.
And here's the big one:
It's all a hierarchy. It is literally objects all the way down.
For all of the same reasons that virtually everyone is now using REST with JSON to implement system interfaces, the same benefits can be realized For any software object, including internal components, packages, namespaces, classes, files, configuration, tables, records, fields, properties, databases, etc.
And the icing on the cake:
It's easy to implement. In fact, it is easier to implement that what we have been doing.
Transparency
"It's gotta be transparent to let the light through!"
Proper implementation of this technique first involves a more complete understanding of what we are actually doing when we implement a JSON REST service, which is just an implementation of Transparency.
Transparency is:
The ability of an object to allow its inputs to be passed through to its outputs – even when the inputs change unpredictably.
Transparency allows objects to be "transparent" to data. This is extremely desirable in multiple ways, allowing the object:
- To be maximally immune to changes in the data
- To ignore data that it is not concerned with
- To be backward compatible with new data
- To implement forward compatibility with unimplemented data changes
- To add information to the data without repercussionReduce or eliminate interfaces that it needs to "know about"
As a result of these benefits:
- Many custom objects in a standard software stack can be eliminated (in exactly the same way that we can eliminate a custom file-format - and most of its associated supporting custom software - by using JSON or XML).
- Changes in one part of the system have less impact on the rest of the system
- Changes to data structure may require little or no software changes. For example, adding a column to a table would make its data immediately available in a view - without any software changes.
- Dependencies are reduced
- Maintenance is reduced
- System reliability is improved
- Time-to market is improved
Implementation of Transparency
There are a number of ways to implement transparency in any system. In all cases, we gain the benefits of decoupled components, improved extensibility, better accommodation of change, and elimination of custom data formats, custom objects and/or custom interfaces.
Transparency in interfaces is being adopted rapidly. It is likely that you are familiar with one or more of the following implementations:
- Exposing interfaces (Such as an external REST/JSON interface.)
- Internal system interfaces (Such as a microservice architecture implemented as REST/JSON)
- Configuration (Such as an XML configuration file.)
- Component interfaces Various open-source or commercial component interfaces exist, such as JCR, JMQ, Database interfaces, Message queues, etc. that use JSON or XML as the payload)
- As a field (such as as a JSON column in a database)
The Universal DTO
One of the most effective ways of implementing transparency is by replacing DTO's (data transfer objects) with a universal DTO (such as some form of JSON object).
Custom DTO's represent a particularly insidious form of technical debt and dependency.
This is because they were once absolutely essential for bridging the gap between proprietary interfaces and your own internal environment, and preventing a complicated proprietary interface from "poisoning" your pristine internal systems. Because of this, we were all taught to make vast numbers of them.
Sadly, it is even common practice for DTO's to fully surround a transparent interface (such as a REST/JSON interface) and thereby nullify much of the benefit of using JSON in the first place.
Consider the following standard client/server stack:
Notice that data is being supplied to, and consumed from, the REST API using DTO's. This means that any changes to the data structure will also require changes to these two DTO's. This situation can be easily avoided, and the system simplified, by replacing the custom DTO's with a universal DTO, such as a JSON Object.
After all, data being supplied by the server needs to be converted from the DTO to JSON anyway. Likewise, on the client side, the data needs to be converted yet again into a DTO.
In their purest form, DTO's represent just data - especially if you have done a good job of keeping them separate from your entities (models). As pure data, they are virtually name/value pairs, becoming little more than technical debt (in the form of maintenance), and are a major impediment to implementing transparency.
A Better Way
A simpler (and transparent) approach is to simply create the JSON object on the server, transmit it to the client, which then passes it on to the business logic and the view. No need to convert it twice - both to, and from, custom classes. In this way, these custom DTO's can be eliminated, while at the same time improving the transparency, simplicity, and speed of the system:
But we shouldn't stop there. The business logic on each side can pass the Universal DTO through. Allowing other objects unimpeded access to the data:
Now the client view is finally receiving unobstructed access to the data. On the server, the other DTO is no longer needed, since the universal DTO fulfills the same function. This further simplifies and improves our system:
Also, the result of a query (a data set) is already delivered from the database in a universal form. (As long as we don't prematurely trap our data in objects via a high-maintenance framework like Hibernate or Entity Framework). It is a simple exercise to convert the result-set to JSON (in fact, many databases will already do this for us):
We have now achieved a transparent stack. Not only is it simpler, more efficient, easier to understand, and easier to maintain. It also fully implements transparency. This means that a column added to the table is immediately available to be used in the view - with no code changes needed in between.
A Light at the End of the Tunnel
Transparency is a powerful tool for reducing the complexity of your system by reducing both dependencies and entities, while at the same time improving performance and making future change easier to implement.
The best implementations of transparency:
- Implement a universal DTO object (UDTO)
- Implement the UDTO as not just a dictionary or a map, but rather as an encapsulation of JSON or other hierarchical collection.
- Use the UDTO for many major internal and external decoupled interfaces
- Allow many objects to be instantiated using a UDTO
- Manage configuration using a UDTO
Next Time
In my next post, we'll explore a number of techniques for extending the management of complexity and dependencies to teams and processes.